The original daily_test trigger and schedule in daily-tests.tf must be filtered to exclude var.kueue_migrated_tests to prevent duplicate triggers from being created and run in the old project. Currently, both the old and new triggers will be created for slurm-gcp-v6-rocky8 because list_tests_midnight still returns it.

If the GKE job fails to schedule or encounters a terminal error (such as ImagePullBackOff or invalid service account), the pod may remain in Pending or another non-running phase indefinitely. Since there is no timeout in this loop, the Cloud Build pipeline will hang until the 24-hour timeout is reached, incurring significant costs. Consider adding a reasonable timeout (e.g., 20 minutes) to this loop.

If the GKE job is deleted externally (e.g., by an administrator or another process), kubectl get job will fail and return a non-zero exit code, leaving both SUCCEEDED and FAILED empty. This will cause the loop to print "Logs disconnected but job is not finished. Reconnecting..." and loop infinitely. Check the exit status of kubectl get job to handle deletion gracefully.

This backup file slurm-gcp-v6-rocky8_old.yaml should be removed from the repository. Keeping old/backup versions of files clutters the codebase and can cause issues with automated test discovery tools (like list_tests_midnight) which might scan all YAML files in this directory and attempt to provision duplicate triggers.

Using static file paths in /tmp (like /tmp/zone_export.sh and /tmp/zone_output.log) can lead to conflicts or security issues if multiple processes run in the same environment. Use mktemp to generate unique temporary files and ensure they are cleaned up on exit.

ZONE_EXPORT=$(mktemp)
ZONE_OUTPUT=$(mktemp)
trap 'rm -f "$ZONE_EXPORT" "$ZONE_OUTPUT"' EXIT

while true; do
	# Run the script in a subshell, streaming stdout and stderr to the console and a log file.
	# To extract the exported variables, we have the subshell write them to a file.
	set +e
	(
		set -e
		source /workspace/tools/cloud-build/find_available_zone.sh
		# If it succeeds, these lines will execute and save the exports.
		echo "export ZONE=\"${ZONE}\"" >"$ZONE_EXPORT"
		echo "export PROVISIONING_MODEL=\"${PROVISIONING_MODEL}\"" >>"$ZONE_EXPORT"
	) 2>&1 | tee "$ZONE_OUTPUT"

	EXIT_CODE=${PIPESTATUS[0]}
	set -e

	if [ "$EXIT_CODE" -eq 0 ]; then
		source "$ZONE_EXPORT"
		break
	else
		# Check if the failure was specifically due to zone capacity
		if grep -q "Couldn't find a zone to deploy" "$ZONE_OUTPUT"; then
			echo "--- RETRYING in 5 minutes... ---" >&2
			sleep 300
		else
			echo "--- FATAL ERROR: find_available_zone.sh failed due to a configuration or system error. Exiting. ---" >&2
			exit 1
		fi
	fi
done